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ABSTRACT 

Here we present a Bayesian formalism for the goodness-of-fit that is the evidence for a 
fixed functional form over the evidence for all functions that are a general perturbation 
about this form. This is done under the assumption that the statistical properties of 
the data can be modelled by a multivariate Gaussian distribution. We use this to 
show how one can optimise an experiment to find evidence for a fixed function over 
perturbations about this function. We apply this formalism to an illustrative problem 
of measuring perturbations in the dark energy equation of state about a cosmological 
constant. 
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1 INTRODUCTION 

Finding an objective Bayesian measure for a goodness of fit 
of a function to some data is problematic because when one 
calculates a Bayes factor (the ratio of evidences for partic¬ 
ular models) the assumption of at least two models must 
be made. To circumvent this assumption one can ask what 
the evidence for a particular (singular) model is over all 
possible perturbations about that model. We present such 
an evidence ratio here, under some assumptions, which is 
calculated using a path-integral methodology (an extension 
of the formalism described in Taylor & Kitching, 2010 and 
Kitching & Taylor, 2010) that marginalises over all func¬ 
tional perturbations about a fixed function. This then pro¬ 
vides goodness of fit that is the evidence for the fixed func¬ 
tion over all other functions. To achieve this we assume that 
the statistical properties of the data can be modelled by a 
multivariate Gaussian distribution. Using this ‘path-integral 
evidence’ we also define a utility function that includes a 
path-integral over the set of perturbed functions and future 
data, this can be used to calculate the expected value of 
evidence for a fixed function over perturbations about that 
function for a future experiment. 

We apply this methodology to the case of determining 
if cosmological experiments can determine the true dark en¬ 
ergy equation of state, as a function of redshift, w(z), from 
perturbations about this function. We use as a simple exam¬ 
ple an experiment that aims to measure w(z ) from changes 
in the Hubble parameter. This is a particularly important 
question because dark energy accounts for the majority of 
the mass-energy content of the Universe and its nature is 
entirely unknown; explanations include proposals to modify 
general relativity on cosmological scales and the addition of 
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new fundamental fields, amongst others. The strongest pre¬ 
diction for w{z) is that it has a value of w(z) = — 1 for all 
redshifts, which would indicate that dark energy is a vac¬ 
uum energy, or additional gravitational constant known as 
the ‘cosmological constant’. Measuring any deviation from 
this functional form, at any redshift, would necessitate the 
need for new physics beyond the standard models of cos¬ 
mology and/or particle physics. This article is arranged as 
follows in Section 2 we present the general methodology, in 
Section 3 we present an application to dark energy experi¬ 
ments, in Section 4 we present conclusions. 

2 METHOD 

We begin with a function that depends on a variable x and 
some parameters 9, that has a fixed (fiducial) functional 
form /f(*|#o), where the parameters take particular values 
do . We perturb this such that any function could be written 
as a perturbation away from the fiducial 

f(x) = f F {x\9 0 ) + S f (x). (1) 

These functions exist in a set M that contains the fiducial 
function, and all perturbations about that function. We can 
write a log-likelihood for the parameters 9 in the case that 
the data can be modelled by a Gaussian distribution 

—2£ = AD T Cg 1 AD + In |C 0 | (2) 

where AD = fi(9) — D is the difference of the mean, that is a 
function of the parameters 9, and the data D and we define 
a covariance Co- We denote the Fisher matrix for the free 
parameters 9 as Fee calculated using the covariance Co- 
From Kitching & Taylor (2011) we can write a likeli¬ 
hood for the parameters 9 accounting for a path-integral 
marginalisation over the functional behaviour of Sf(x), by 
changing the covariance such that 
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Cm = [Co 1 - Co'TCq 1 ]- 1 


( 3 ) 


where 


T = 


dxdx' 


Sfj, 


SS f (x) 


F s f \ f {x,x) 


Sfi 

JsJ{T) 


( 4 ) 


the mean is now a function of 8 and 8/(x), and Fs f s f (x,x') 
are the elements of the functional Fisher matrix for the per¬ 
turbations 5f marginalised over the parameters 9. This is the 
‘self-calibration’ case discussed in Kitching & Taylor (2011) 
where the prior in function-space is a flat top hat; for a 
detailed discussion of flat functional priors see Bornkamp 
( 2011 ). 

In the case that a functional prior is included over the 
perturbed function-space Sf(x) the covariance reduces (via 
the Woodbury matrix identity) to 


Cm = Co + G 


( 5 ) 


where 

e ^ (6) 

where Cp is the covariance of a Gaussian functional prior. 
Either the flat or Gaussian-prior cases (equations 3 and 5) 
can be used to construct a new Fisher matrix for the param¬ 
eters 8 , now taking into account the marginalisation over 
perturbations <5/ about the fixed function. As discussed in 
Taylor & Kitching (2010) the flat prior case is different to a 
Gaussian prior with large variance, they have different func¬ 
tional weighting that means the latter allows for the use of 
the Woodbury identity whereas the former does not. 

We can now extend the Bayesian evidence formalism 
of Taylor & Kitching (2010), to include the marginalisation 
over the perturbed functional behaviour. Referring to Tay¬ 
lor & Kitching (2010, equation 51) we write the evidence, 
assuming that the data can be modelled using Gaussian dis¬ 
tributions, including marginalisation over all perturbations 
about the fiducial function 

In E m = AD t CmAD -(- In | Cm | 

+ 2 \n{V M y/\F**\) - N m In 2 tt (7) 

where Cm is defined either in equation (5) or (3) depending 
on the prior. V M is the volume over which the prior distri¬ 
bution is defined. F M is the Fisher matrix for the fixed pa¬ 
rameters after marginalisation over the pertubed case, this is 
defined as the Schur complement of the full Fisher matrix F 
i.e. F f = Fij — Fi a F~pFpj, where a and /3 here represent 
function elements of the total Fisher matrix, and Roman 
letters represent parameters of the fixed function. We can 
also write the evidence for the fixed function by referring to 
Taylor & Kitching (2010, equation 54) 

ln£° = AD T Cq 1 AD + In |C 0 | 

+ 2 ln(V° v / l^°T) - No In 2 t r, (8) 

where Co is the covariance not marginalising over the per¬ 
turbations Sf(x), and F° is the part of the Fisher matrix 
for the parameters of the fixed function only. V° is the vol¬ 
ume over which the prior distribution is defined, in the fixed 
function case. No = No — Np is the number of degrees of 
freedom where No is the number of data points and Np is 
the number of free parameters. In the case of the perturbed 
case Nm —> —oo. We address the infinity below. 


By combining the expressions for the evidence in the 
fixed plus perturbed (F + P) case f(x) = f F (x) + Sf(x), and 
the fixed case f(x) = f F (x) only (F), we can write a Bayes 
factor that quantifies the ability of the data to distinguish 
these models 

B f/(f+p) = In E° — In E m 

= AD t (Co 1 - C^AD 

+ lnlCoCM 1 !. (9) 


This is the evidence for the fixed model over the fixed plus 
the perturbed case. 

The final terms in the evidence, the Occam factor, do 
not appear in this evidence ratio. This is because, in the 
perturbed case the parameter space volume tends to infin¬ 
ity. There are two motivations that could be used to justify 
the removal of these terms. Firstly the removal of the Occam 
term can be considered as a practical measure. The Bayes 
factor would then not account for the Occam factor, and is 
therefore not a true Bayesian measure, but does still capture 
the change to the posterior caused by the increase in covari¬ 
ance as a result of marginalisation over the larger parameter 
(function) space. 

Alternatively it can be justified by modifying the prior 
volume V M in the perturbed case such that in the two dif¬ 
ferent models this term cancels. This is achieved by scaling 
the volume V M (a hypersphere with radii scaled using the 
ratio of the expected Fisher matrices), over which we allow 
the prior in the perturbed case to be defined, by the prior 
volume of the unperturbed case 


IA’ 


V 0 (2n) (NM ~ No)/2 



pM 


( 10 ) 


The above expression is the volume of the hypersphere V M 
scaled from V°. Note that the integral over prior (outside 
this volume) can tend to infinity but that in this case, be¬ 
cause we always scale with respect to V° in taking the evi¬ 
dence ratio, the Occam factor terms cancel to zero. This is 
equivalent to setting a prior on the parameter volume, as a 
result this metric will not be sensitive to penalties induced 
from an unexpected increase in volume. Without some jus¬ 
tification for the removal of the Occam factor the evidence 
ratio would result in an infinite result (the marginalised case 
having an infinite number of degrees of freedom), but we 
note that the scaling is convergent, and so asymptotically 
should behave correctly. 


2.1 Expectation Value 

We now explore the expectation value of the above statistic, 
and show how we can forecast this Bayes factor for experi¬ 
mental optimisation. 

For forecasting purposes we need to integrate over the 
future data vector, as described in Amara & Kitching (2010) 
and Trotta et al. (2011) where a an ‘expected utility’ or ‘fig¬ 
ure of merit’ is defined as an integral over parameters (or 
functional integrals in our case), and data vectors. The util¬ 
ity function here case is the log Bayes factor for the fixed 
model over the perturbed case 6f/(f+p), given in equa¬ 
tion (9), where we have already performed the integrals over 
functions and parameters. The final integral over the data 
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vector arrives at an expression for the expected evidence 
ratio 

(lB f/(f+p) ) = l-Tr(C 0 C^) + ln|C 0 C'^ 1 |. (11) 

We recognise the two terms above as i) the fractional change 
in the covariance, ii) the fractional change in the volume of 
the covariance due to the functional marginalisation. This 
expression now quantifies the expected ability of an exper¬ 
iment to distinguish a fixed function f F {x) from the fixed 
plus perturbed case f F (x) + Sf(x). 

Interestingly, this in fact has a similar form to the 
Itakura-Saito distance that is a measure of the ‘perceptual 
difference between an original spectrum P and an approx¬ 
imation P of that spectrum’, of use in signal processing, 
but here we derive this from first principles. More generally 
equation (11) is similar to the the Kullback-Leibler diver¬ 
gence Dkl(p1,p2) = J dxpi(x)ln(p 2 (x)/pi(x)), which is a 
measure of the difference between two distributions, in our 
case the probability of the true function and the probabil¬ 
ity marginalised over all functions. The Kullback-Leibler di¬ 
vergence was investigated as a utility function or ‘figure of 
merit’ for cosmology in Amara & Refregier (2014). How¬ 
ever this only looked at simple functional changes, while we 
generalise to the continuous functional case and derive from 
first principles. 

We can also calculate the evidence for the fixed model 
over the perturbed case 

(B f/p ) = — ln[e“ <8F /T+P)> - i], (12) 

Here we assume that the evidences for the fixed and per¬ 
turbed models are additive, p(D\F + P) = p(D\F)+p(D\P), 
where the probability p(D\F + P) corresponds to the ‘prob¬ 
ability of the fixed OR perturbed’ case (i.e. any function). 
This is true because F and P are mutually exclusive; where 
here we define the perturbed model-space as all functions 
except the fiducial case (another way of formulating this is 
that fiducial function F is a special case within the space 
of all functions, in that its prior volume is zero). This is 
an expression for the expectation value of the evidence for 
the true function over perturbations away from that true 
function. To calculate this we make the approximation that 
<■ B F/(F+P) )=ln[(E 0 /(E° + E M ))]. 

This statistic can be used to optimise experiments so 
that they will have the ability to distinguish a fixed func¬ 
tion from peturbations about that function. In practice one 
would fit a free functional form to the data (see e.g. Critten¬ 
den et al., 2011). For calculation purposes one must assume 
a ‘fiducial’ model for the fixed function f F (x) around which 
to calculate the Fisher matrices and covariances. However 
we note that under the assumption that the data can be 
modelled by a multivariate Gaussian the expressions above 
are independent of the choice of fiducial function (see Taylor 
& Kitching, 2010). 

3 EXAMPLE APPLICATION 

We now apply this methodology to the dark energy equation 
of state w(z), where we have a model that includes a fixed 
function of w F (z) and we include perturbations about this 
model 

w(z) = w F (z) + 5 w (z), (13) 


where the function is over redshift 2 . We look at an illustra¬ 
tive example in this paper as a proof of concept, where we 
consider a measure of the Hubble parameter H(z) which is 
related to w(z) via the follow equation 

H(z) = H 0 [n M (l + z) 3 

+ «x(l + zffo d *' [1 +» ( *' )1/(1 +*')] 1/2 (14) 

where Ho is the current rate of expansion, Hm and Q.x are 
the dimensionless matter and dark energy densities respec¬ 
tively and w(z ) is the dark energy equation of state. We 
consider a hypothetical experiment that measures H(z) in 
ten redshift bins between redshifts 0 < 2 < 2 with an accu¬ 
racy of a h in each redshift bin. We use a fiducial cosmology 
that assumes Ho = 70kms -1 Mpc -1 , 17 m = 0.3, fix = 0.7 
and w F (z) = —1 V 2 . We assume a functional prior on w(z ) 
that is diagonal in redshift Ci :> s w(z - ) s w(z , ) = a P S^.,. In Figure 
1 we show an example where the measurement on H(z) has 
an error bar of = 10 —3 , all errors have the same units as 
H. 

It is worth spending some time examining the generic 
behaviour seen in Figure 1. The theory prior is a func¬ 
tional prior around the fixed function. For the Bayes factor 
(Bf/(f+f)) what we find is that as the theory prior ap¬ 
proaches infinity the Bayes factor tends to minus infinity, 
this means that data would favour a perturbed model. As the 
theory prior tends to zero the Bayes factor (£?f/(f+p)} —> 1, 
as expected from equation (11), this means that if one per¬ 
formed this experiment with a small theoretical prior there 
would be no change in evidence (Bf/(f+p)) i.e. with this 
prior it would not be worth doing this experiment because 
the space of models tested is already well constrained by 
the prior, and the experiment is expected to favour the fixed 
function. 

The Bayes factor ( B F / P ) has an interesting and impor¬ 
tant behaviour. When (£?f/(f+p)} = 0 there is a singularity 
where the expected evidence for the fixed model tends to 
infinity. This boundary is of particular importance. If one 
performs an experiment with a pre-experimental theoreti¬ 
cal prior smaller than a particular value (leftward of this 
singularity in Figure 1) then the experiment is likely to re¬ 
turn no change in evidence over that prior, and is likely to 
favour the fixed function. However, if one performs an exper¬ 
iment with a pre-experimental prior that is larger then the 
experiment is likely to return a change in evidence (either 
a weaker preference for the fixed function, or for the per¬ 
turbed case). In colloquial terms this singularity demarks, 
for a given expected experimental error, and a given pre- 
experimental theoretical prior, whether the experiment is 
‘worth doing’. 

To explore this concept further we show in Figure 1 the 
case of two different expected error bars on H(z). Consider 
in this Figure the case that before doing the experiment 
the theoretical prior about w(z) was ap = 1, in this case 
it would be worth doing an experiment with an expected 
error of a h = ICE 3 but it would not be worth doing an 
experiment with a h = 5 x 10~ 3 . 

Furthermore as the error bar decreases the expected 
evidence for of finding a perturbation increases (( B F / P ) be¬ 
comes more negative). If one is rightward of the singular¬ 
ity in Figure 1 then an experiment may be worth doing, 
but it can be highly likely that it will find no evidence 
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Figure 1. Left: The expected evidence (Bp/p) (fixed vs. perturbed) and (,Bf/(f+P)) (fixed vs. all) as a function of the pre-experimental 
theoretical prior. This is for the illustrative example of measuring a perturbation about a fixed value of the dark energy equation of state 
w(z) = —1 using measurements of the Hubble parameter. Shown are expected evidence for an expected error on the measurements of 
a H = 10 -3 . Right: The expected evidence (Bp/p) (fixed vs. perturbed) for two different values of the expected error on H. 


for a perturbation. As the experimental error bar decreases 
one becomes more likely to find evidence for a perturba¬ 
tion over the fixed case (negative values of (Bp/p)). There 
is a point where the evidence of the fixed function becomes 
zero (Bp/p) = 0, this occurs when (jBf/(f+p)) = — ln(2) i.e. 
that there is equal evidence for the fixed and perturbed case 
«Bf/ ( p+p)) = In [Ep/(E f + E P )] = ln(l/2)). 

We explore this in Figure 2 where the expected evidence 
(Bp/p) is shown as a function of the expected experimen¬ 
tal error <jh- When the expected error is larger there is no 
expected change in evidence due to the experiment. As the 
expected error decreases there is a regime where the exper¬ 
iment becomes worth doing - but the result is expected to 
confirm the non-perturbed model (i.e. given the theoretical 
prior the data is unlikely to provide evidence against the 
non-perturbed model). However when the expected error is 
small enough, in this case uh < 10~ 3 , the experiment is 
worth doing and is expected to return evidence for a per¬ 
turbed model over the non-perturbed case - if such a per¬ 
turbation exists. 

4 CONCLUSION 

Here we present, under the assumption that the data can 
be modelled by a multivariate Gaussian distribution, a 
Bayesian goodness-of-fit which compares the evidence for 
a fixed functional form to the total evidence for all func¬ 
tions that are not the fixed functional form. We then define 
a utility function using this metric that marginalises over all 
functional perturbations about a fixed model. 

We apply this using an illustrative example of determin¬ 
ing the dark energy equation state through measurements of 
the Hubble parameter. We find that the expected evidence 
provides a clear metric, given a pre-experimental theoret¬ 
ical prior and expected error bar, that can quantify if an 
experiment is likely to produce a change in the evidence; 
answering the question is an experiment ‘worth doing’ ? We 
find that as the expected error bar for a planned experiment 
decreases the expected evidence for finding a perturbation 



Figure 2. The expected evidence (Bp/p) (fixed vs. perturbed) 
as a function of the expected error bar on the Hubble parameter 
rr p : here we fix the pre-experimental theory prior to crp = 1. 

can become significant, and we suggest that this metric may 
be useful in experimental design. For our illustrative exam¬ 
ple we show that in order for an experiment to expect to 
find strong evidence for a perturbation about w(z) = — 1 
a measurement of the Hubble parameter only would need 
to have errors of (jh < ICR 3 over ten redshift bins between 
0 < z < 2. 

This formalism should be applicable to any case of ex¬ 
perimental design that seeks to maximise the expectation of 
finding perturbations about a functions, and as such can be 
used as a model-independent tool for experimental optimi¬ 
sation. 
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